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Summative Evaluation of PLATO Computer-Managed Learning 

in the 

Nursing Assistant Program 
April, 1987 

EXECUTIVE SUMMARY 

Following the recommendations of an external consultant 
(Montgomerie, 1985) , the Nursing Assistant Program and the 
Division of Research and Evaluation, Alberta Vocational Centre, 
Edmonton, designed and implemented pilot projects to test the 
Control Data PLATO system, and PLATO Learning Management (PLM) 
software, for Computer-Managed Learning (CML) in the Nursing 
Assistant Program. Positive results from the pilots resulted in 
plans for CML implementation in the Nursing Assistant Program in 
September , 1986, with one class of Nursing Assistant Program 
students . 

Successes with the September, 1986, class led to application 
of PLATO CML with all subsequent Nursing Assistant Program regular 
and Refresher students in the 1986-1987 academic year. 

Conclusions of the implementation project were as follows: 

Conclusion 1: All but one recommendation of the Montgomerie 

Report was met fully and one recommendation was met 
partially in the PLATO-based CML implementation 
projects. The un implemented recommendation , 
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reiterated in this study, was for an institutional 
instructional design group (with project management 
responsibilities) . 

Conclusion 2: The project development process devised for the 
project and used from December , 19 8 5 , to 
November, 1986, successfully achieved the project's 
initial objectives . 

Conclusion 3: Training and scheduling of Nursing Assistant 

Program students was effectively and efficiently 
performed. (While experience showed that all 
classes, including Refresher, could be accommodated 
with existing resources, there was evidence that 
resources were somewhat strained by the level of 
usage . ) 

Conclusion 4: Staff training achieved the goal of making 

instructors comfortable with CML and capable of 
functioning as instructors in the PLATO 
environment . 

Conclusion 5: The student schedule of four hours of PLATO access 
per day, while adequate, was perceived as too 
constraining by some students. It was suggested 
that evening hours be initiated as soon as 
possible. 

vi 
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Conclusion 6: Present posttests were suitable for present 

purposes. Further expansion of the item bank by 
one-third was recommended. 

Conclusion 7: Students were quickly capable of using the PLATO 

system effectively, and reported finding PLATO a 
useful and enjoyable experience. 



Conclusion 8: 



Conclusion 9: 



Incorporation of CML resulted in actual cost 
savings of 1/3 to 1/2 over manual, instructor- 
managed systems for on-site delivery of the 
program. 

The results of the CML project showed great promise 
for future expansion and refinement, both with 
regular on-site clientele and in off-site, distance 
delivery locations . 



Conclusion 10: Further studies were recommended to answer remain- 
ing research questions concerning CML in this and 
other AVC programs. 
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SUMMATIVE EVALUATION OF PLATO COMPUTER-MANAGED LEARNING 

in the 

NURSING ASSISTANT PROGRAM 
Alberta Vocational Centre, Edmonton 
April, 1987 
P. Faliy 

Director, Research and Evaluation 

Overview 

In a study of the instructional computing needs of the 
Nursing Assistant Program conducted in 1985, the following 
conclusions were reached (Montgomer ie, "Computer-Based Learning 
Investigation: Final Report", 1985); 



Recommendation #1: Alberta Vocational Centre, Edmonton 

should incorporate Computer-Based Learn- 
ing in the Nursing Assistant Program. 

Recommendation #2 : Implementation should be phased in . 

(This recommendation gave priority to 
Computer- Managed Learning, with 
Computer -Ass is ted Learning delayed 
"until the CML system is in place and 
working well" ( Ibid . , p. 85). 

Recommendation #3: A team approach should be used. (Skills 

to include "content , instruct ional 
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design, knowledge of the CBL system, and 
management 11 (ibid ) . 

Recommendation #3A: A minimum half-time CBL Coordinator 

should be appointed ( Ibid . , p. 86). 

Recommendation #3B: A centralized instructional design group 

should be established. (To provide 
instructional design expertise to the 
whole institution) . 

Recommendation #4: Computing facilities should be contract- 
ed from some other institution. 

Recommendation #5: The PLATO CBL system should be used. 

Recommendation #6: There should be no integration with the 

Student Information System at this time. 
(Computer-Managed Learning records 
should be kept "as day-to-day operation- 
al information" . ) 

The present report describes efforts from December, 1985, to 
January, 1987, to implement the above recommendations. 

Background 

In autumn, 1985, preparations for implementation of the 
recommendations of the Montgomerie report commenced* In November , 
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1985, a Priority Employment Program (PEP) Aide was trained to use 
PLATO Learning Management (PLM) and commenced inputting post test 
items, using existing test items developed for paper-and-pencil 
testing . 

The first pilot project, comprising Modules 20 to 23 and six 
student participants, was conducted from January 13 to 17, 1986; 
the second, comprising Modules 66 to 69, and 74 to 76, with 5 
students, occurred February 10 to 14, 1986. 

As the inputting of posttest items went quickly and the 
results of the pilot projects were very positive, plans were made 
to conduct a third pilot commencing March 25, 1986, with a full 
class of 30 students over a complete level (Level 1) of the 
curriculum. Because telecommunications equipment was not install- 
ed in time, however, this pilot did not occur. Nevertheless, on 
the basis of previous successes, the decision was made to proceed 
with planning for full-scale implementation of CML in September, 

1986, and the time from March to September was used to develop a 
detailed description of Computer Managed Learning components (see 
Attachment C ) . 



Methodology 

Evaluation of the Project was 
during which the first (September) 
1986, to May, 1987 (see Evaluation 
withdrawal by the Nursing Assistant 

ERIC 11 



planned to occur over the term 
class was enrolled, September , 
Plan, Attachment A). However, 
Program of the services of the 
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CML manager (see below) resulted in a truncated evaluation and 
deletion of several research objectives. The effective term of 
the evaluation was thus December, 1985, to January, 1987, with 
emphasis on the first part of the implementation phase, 
September, 1986, to January, 1987. 

Evaluation objectives which were addressed during this period 
include the following: (see Attachment A): 

1. Train and schedule the September Nursing Assistant class in 
use of the PLATO-based module post tests and CML package. 

1.1 Because of the success of this implementation, all 
classes since September, 1986, have used PLATO CML. This 
report will provide information on some of the experi- 
ences of the September, October, and December, 1986, and 
February, 1987, classes. 

2. Observe students in the CML testing environment and make 
adjustments to training, scheduling and materials as needed. 

3. Document some of the perceived effects of CML from the 
instructors ■ perspectives . 

4. Document some o? the perceived effects of CML on students' 
attitudes . 

5. Define the roles of team members: Project Manager, Project 
Coordinator, paraprof essionals , content specialists, instruc- 
tional designers, CML specialists. 
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6. Document components of the total CML package and their costs, 
in comparison with alternatives. 

Data were gathered throughout the project by the Division of 
Research & Evaluation, and, until her reassignment, by the Nursing 
Assistant Program CML Project Manager. PLATO records provided 
information on times and patterns of use; observation and student 
comments were sources of anecdotal information. Questionnaires 
were administered in November and December, 1986, and in February, 
1987, to assess students* initial and later attitudes (Attachments 
E and F) , and students and staff participated separately in group 
interviews in February, 1987. 

Findings 

Objective 1 ; Scheduling and training of students in Computer- 
Managed Learning (CML) 

Pilot project data indicated that students required 8 to 10 
minutes of terminal time per module posttest, not including retest 
time. A schedule was developed providing 15 minutes per session 
per student, and estimates of terminal time required for each week 
of the year were made (see Attachment B) . Projections indicated 
that i f students used 15 minutes per test there would be 10 weeks 
in the year when students would require more than 3.5 hours of 
daily access per terminal in order to complete testing for the 
week. There would thus be 10 weeks in which resources would be 
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severely strained. Table 1 compares actual and projected activity 
levels for the period September, 1986, to January, 1987. 



Table U Projected and Actual PLATO Activity Levels, Nursing Assistant Program, Septeaber, 1986, to January, 1987^ 

ftctive I of Total Average Terminal 

Students Sessions Hours Hours 



Month 


Projected 


Actual Z Diff.t 


Projected 


Actual Z Diff.t 


Projected 


Actual I Diff.t 


Projected 


Actual Z Diff. 


Septeaber, 1986 


30 


31 


1031 


720 


775 


1081 


156.2 


145.7 


932 


1.24 


1.16 


94Z 


October 


60 


75 


125Z 


570 


560 


981 


156 


129.3 


83Z 


1.3 


1.08 


83Z 


Hovecber 


60 


74 


1231 


960 


1035 


1081 


256.5 


270.2 


105Z 


2.25 


2.37 


105Z 


Deceaber 


90 


104 


116Z 


990 


682 


69Z 


211.7 


82.6 


39Z 


2.08 


0.81 


39Z 


January, 1987 


90 


89 


99Z 


1560 


1045 


671 


412.7 


237.7 


5BZ 


3.2B 


1.89 


58Z 


TOTAL/AVG 


66 


74.6 


1131 


4800 


4097 


851 


1193.1 


865.5 


73Z 


2.03 


1.462 


72Z 



(Actual as percent ol projected. 



The major conclusion supported by Table 1 is that while the 
number of students using PLATO CML was 13% higher than expected 
(due primarily to inclusion of 17 refresher students in November, 
1986), the number of sessions was 15% less, the total number of 
hours 27% less, and the average number of terminal hours per day 
28% less than projected. The probable explanation for the 
discrepancy between actual and projected totals is that the pilot 
projects on which the projections were based represented student 
performance early in the learning curve; when students became more 
comfortable and proficient with the system their efficiency 
increased and the time required decreased. (If it is true that 
estimates were approximately 25% too generous, it is possible that 
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12 minute test sessions ccMld be initiated. However, as will be 
noted below, students would not support such a decision) . 

Students 1 training in use of PLATO CML consisted of one in- 
class orientation to CML concepts such as self-pacing and 
self-direction, and to basic rules for the course (see Attachment 
D), followed immediately by a hands-on session at the PLATO 
terminal in which students worked in pairs or trios to complete a 
demonstration module. After orientation students were free to 
schedule themselves for testing on the first modules of Level 1. 

The CML lab was available to the students at the fo] -'ing 
times : 



7:30 - 8:30 a.m. 
12:00 - 1:00 p.m. 
2: 30 - 4: 30 p.m. 

Daily scheduling was by reservation of a 15 minute testing 
block using a sheet posted outside the terminal room. Ordinarily, 
students were not permitted to reserve two consecutive test 
blocks, nor to reserve time more than one day in advance. 
Specific times of the day were sometimes reserved for individual 
classed to assure that students within each class would have 
sufficient time to complete required tests. Students were asked 
to cancel their reservations if unable to keep them; to minimize 
the impact of failed appointments a policy was adopted allowing a 
two-minute grace period, after which a waiting student could sign 
on . 
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Initially, the CML lab was supervised at all times by an 
instructor. Later, instructional assistants took part as well. 
Stiil later, unsupervised times were initiated, during which an 
"honor system" was invoked . (Due to the early termination of the 
research phase of the project, causes of problems with unsuper- 
vised CML use, though detected, were not assessed. ) 

Objective 2 : Adjustments to the CML Environment 

Training Adjustments . No major gaps in student CML training 
were identified. Individual students received more attention if 
they appeared to require it. Overall, it appeared that the train- 
ing system as described in Attachment D was adequate. 

Scheduling . In December, a decision by the Nursing Assistant 
Program to close the CML lab from 12:00 to 1:00 made CML unavail- 
able during the period most heavily used and preferred by 
students. The problems caused for students by this decision were 
partially addressed by assignment of staff from the Division of 
Research and Evaluation, permitting the lab to open from 12:30 to 
1:00 p.m. daily. After further discussion with the Nursing 
Assistant Program, the deci sion was reached to resume Nursing 
Assistant Program supervision of the lab from 12:00 to 12:30 each 
day, beginning in February, 1987. 
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Student reaction to closing of the noon hour time period was 
immediate and definite, and the issue of scheduling was mentioned 
in all four class interviews. Students made these points: 

1 . In general , the schedule mitigated against students who 
could not come early or stay late in the instructional 
day, a condition exacerbated when the noon hour was 
lost . 

2. The noon hour time period was generally felt to be the 
most desirable, as one hour for lunch was thought by most 
students to be too long. 

3. Many students would use evening sessions for CML activi- 
ties if they were available. 

4. Students found the fifteen minute test block to be minimal 
for useful work; they did not believe 12 minutes would be 
adequate . Several students requested longer sessions , 
pointing out the Refresher students had one-half hour time 
blocks in which to complete their testing. 

Materials . Overall, 62% of the allotted capacity of the 
Nursing Assistant Program PLATO test bank is presently used (see 
Table 6). 
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Table 6: Test Bank Description 



Modules 


Bank A 


Bank B 


Total 


Total 


Total 


1 








Bank 


Mastery 


Used 


Used 


1-19 


204 


101 


305 


161 


209 


62 


20-39C 


312 


104 


416 


197 


261 


53 


40-57C 


255 


112 


367 


190 


242 


55 


58-70 


236 


85 


321 


145 


181 


68 


71-B5 


19B 


93 


291 


134 


173 


62 


B6A-90B 


120 


38 


158 


68 


81 


72 


TOTAL 


1325 


533 


1858 


895 


1147 


62 



While no new test items were added to the bank during the 
research phase, errors in test items Jere corrected by Research 
and Evaluation staff and the CML Manager as they were identified. 
Further efforts to identify or create off-line materials to serve 
as instructional prescriptions, essentials of the CML package (see 
Attachment C), remain to be expended. 

Objective 3: Effects of CML on Instructors 

Six instructors involved in the CML project were interviewed 
in February, 1987. As in the student interviews, two areas of 
focus were defined: what benefits were occurring from CML, and 
what changes, suggestions, or problems would they identify. 

Instructors were emphatically positive about CML. Advantages 
included: 

1. Fewer test ing/retesting clerical duties *\ £ instructors 
and consequently more time for instruction - 
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2. Less stress for students in testing. 

3. Less time taken up in irrelevant debate in class about 
test items. 

4. More student responsibility for learning. 

5. (Especially for Refresher students) more efficient use of 
scarce on-site time . 

Instructors were also positive about their own and the stu- 
dents 1 use of performance records produced by PLATO, They felt 
these records kept them well informed and gave students a clear 
sense of their own achievement. 

No serious problems emerged from the staff interview. How- 
ever, these suggestions were made: 

1. Students need to be reminded about "schedule etiquette". 
(Over-staying their time or f ai ling to respect others 1 
times were occasional problems.) 

2. Cheating during unsupervised periods may be/become a 
problem (the purpose of the module post tests as a study 
aid should be reiterated). 

3. Students failing to master a module should be advised to 
study prior to rewriting. 
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Objective 4 : Effects of CML on Students 1 Attitudes and 
Behavior 

The October and December, 1986, and February, 1987, classes 
completed a pretest Computer Adaptation Scale during their 
orientation, prior to any use of PLATO, and a posttest Computer 
Adaptation Scale after completing Level 1, approximately 6 weeks 
later (Attachments E & F), (The September class completed the post- 
tests only). Table 2 shows pre- and post- use rankings of the 
questionnaire's 12 items. 



Table 2: Pre- and PDsttest Rankings of Student Questionnaire Iteas 





Pre-use 


Post-use 


Difference 


Pre 


Post 


I THINK/FOUND USING A COMPUTER 


average 


average 




Rank 


Rank 


FOR TESTING ... 












1. Mould aake/aade le nervous 


2.32 


2.21 


-0.11 


8 


8 


2. Mould be/Mas easy 


2.97 


3.05 


0.0B 


5 


5 


3. Mould be/Mas sloMer than 












paper and pencil 


1.66 


1.37 


-0.31 


12 


12 


4. Mould be/Mas too iapersonil 


1.97 


1.94 


-0.03 


9 


9 


5. Mould aake/aade ae feel too 












isolated 


1.77 


1.6B 


-0.09 


11 


11 


6. Mould be/Mas oore flexible 


2.66 


3 


0.34 


6 


6 


7. Mould be/Mas aore efficient 












use of ay tiae 


3.21 


3.41 


0.2 


1 


1 


8. Mould be/Mas satisfying 


3.01 


3. OS 


0.07 


4 


4 


9. Mould be/Mas frustrating 


1.97 


1.79 


-0. 18 


10 


10 


10. would be/was interesting 


3.2 


3.36 


0.16 


2 


2 


11. would/did iaprove ay 












knoMledge of the subject 


2.52 


2.7 


0.1B 


7 


7 


12. would be/Has enjoyable 


3.13 


3.22 


0.09 


3 


3 



Scale: 4 = Strongly Agree, 3 = Agree, 2 = Disagree, 1 = Strongly Disagree 
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Table 2 shows that the most positive ratings (all of which 
were increased by experienced with PLATO ) were given for these 
items (mean rank in parentheses): 

7. PLATO testing was more efficient use of time (3.41) 

- 10. PLATO testing was interesting (3.36) 

- 12. PLATO testing was enjoyable (3.22) 

- 8. PLATO testing was satisfying (3.08) 

The least agreement was expressed for these items, all of 
which received even less agreement after PLATO experience: 

- 3. PLATO was slower than paper-and-pencil testing (1.37) 

- 5. PLATO made me feel too isolated (1.68) 
9. PLATO was frustrating (1.79) 

- 4. PLATO was too impersonal (1.94) 

- 1. PLATO made me nervous (2.21) 

In summary, experience strengthened students 1 opinions that: 
PLATO was an efficient use of their time, was enjoyable, was 
interesting, and was satisfying; PLATO was not slower than paper- 
and-pencil testing, did not make them nervous, was not too 
impersonal, did not make them feel isolated, and was not frustrat- 
ing. 

(See Attachment G for a class-by-class comparison of question- 
naire results . ) 
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In addition to the questionnaire, students in the September, 
October and December, 1986, and February, 1987, class were inter- 
viewed in class groups regarding their PLATO experiences. Both 
positive elements and suggestions for change were solicited. 

Students reported that the benefits of PLATO CML were: 

1. Testing was available when the student was ready. 

2. There was less stress. 

3 . Instant feedback was helpful . 

4. Self-pacing was helpful. 

5. Review was valuable. 

6. PLATO was orderly, logical, and very friendly. 
Suggestions for improvements included the following : 

1. There should be more terminals. 

2. There should be more testing time (ie., during lab testing 
when students wait to be performance tested). 

3. There should be more privacy (ie., dividers between 
terminals) . 

4. There should be better noise insulation from adjacent and 
classrooms and hallways. 
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5. There should be more access to PLATO during clinical weeks 
for those who wish to come back into the institution after 
work . 

6. Testing blocks should be longer (up to 1/2 hour). 

7. Staff should not watch students over-the-shoulder while 
they are testing. 

8. There should be more opportunity for review. 

9. PLATO should be available earlier and later in the day, 
including evenings . 

Another finding related to student use of CML for self-pacing 
is contained in Tables 3 & 4. 



Table 3: Septepber, 1986, Nursing Assistant Class. 



Survey 


Huaber of 


Behind 




On 


Ahead of 


Date 


Students 


Schedulet 


Schedulet 


Schedulet 






f 


* 


1 


* 


i 


* 


Septeeber 11, 1986 


30 


9 


30* 


IB 


60* 


4 


0.13 


Septesber 18 


30 


11 


37* 


8 


27* 


12 


0.4 


September 25 


30 


3 


101 


9 


30* 


19 


0.63 


October 16 


30 


7 


23* 


9 


30* 


14 


0.47 


October 30 


30 


10 


33* 


11 


37* 


9 


0.3 


Novesber 18 


30 


7 


231 


11 


37* 


12 


0.4 


Noveaber 27 


30 


15 


50* 


9 


30* 


6 


0.2 


Deceeber 11 


26 


5 


19* 


17 


65* 


4 


0.15 


January 2 


26 


8 


31* 


12 


46* 


C 


0.19 


January IS 


25 


5 


20* 


7 


28* 


13 


0.52 


Juii.ary 30 


23 


15 


65* 


8 


35* 


0 


0 


TOTAL/AVG 


28.2 


7.3 


31* 


9.2 


39* 


7.5 


31* 



(According to the daily schedule. 
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Table 4: Noveaber, 1966, Nursing Assistant Class. 



Survey 


Nuaber of 


Behind 




On 


Ahead of 


Date 


Students 


Schi 


>dulet 


Schedulet 


Sche 


dulet 






f 


I 


t 


7 


t 


7 


October 30 


30 


27 


907 


3 


107 


0 


07 


Noveober 7 


30 


9 


307 


5 


177 


16 


537 


Noveaber 20 


30 


2 


77 


8 


277 


20 


677 


Noveaber 28 


30 


21 


m, 


B 


277 


1 


37 


Deceaber 5 


29 


9 


317 


8 


2B7 


13 


457 


Deceaber 18 


30 


5 


177 


11 


377 


14 


477 


January 2 


29 


1 


37 


12 


417 


16 


557 


January 8 


29 


15 


527 


9 


317 


5 


177 


January 23 


28 


8 


297 


17 


612 


3 


117 


TOTAL/AVG 


29.4 


10.8 


377 


9.0 


317 


9.B 


337 



tAccording to the daily schedule. 



Tables 3 and 4 show that students in these two classes main- 
tained different paces as they moved through the curriculum: 
approximately 1/3 were slightly behind the scheduled pace, 1/3 
were on schedule, and 1/3 were slightly (or in some cases consid- 
erably) ahead of schedule. These findings corroborate a major 
contention of Bloom (Human Characteristics and School Learning, 
1976) that, when time is variable but achievement constant (as in 
competency-based learning ) , students wi 11 find a personally 
suitable learning pace which may vary markedly from the group 
average . 
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Objective 6 ; Role of Team Members 

As noted earlier, recommendation 3 of the Montgomerie report 
was that a team approach should be employed in developing 
Computer-Based Learning, to include expertise in "content, 
instructional design, knowledge of the CBL system, and management" 
( Ibid , , p. 85). 

In the pilot projects and the implementation project the 
Director of Research and Evaluation supplied expertise in curricu- 
lum design and the PLATO system, the Nursing Assistant Program 
provided content expertise and day-to-day coordination in the 
person of the Nursing Assistant Program Computer-Managed Learning 
Manager, and management was comprised of representatives of the 
Nursing Assistant Program, Research and Evaluation, and the insti- 
tution's senior administr / tion. 

The experience of this implementation has shown that the 
following tasks are associated with these roles: 

Program Content Expert ; 

1. Develop, revise, and correct test items . 

2. Produce training/orientation schedules and materials for 
students and staff. 

3. Monitor needs for and uses of system records* 

4. Train and supervise paraprof essionals . 
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5, Identify needs for and acquire off-line learning 
resources . 

6. Monitor system student notes and refer/reply, 

7. Monitor system reliability, 

8, Identify evaluation needs. 

Instrxr lonal Designer 

1. Val-idc -e test items. 

2. Develop and monitor system for test item generation, 
testing, review, revision and installation, 

3. Document intended learning goals and objectives, and 
actual outcomes. 

4. Assess impact of CBL on total learning system and suggest 
action to maximize results. 

5. Plan and conduct evaluation. 

6. Disseminate outcomes. 

CBL Expert 

1. Act as PLATO Account Director (create groups, sign-ons, 
set accesses, etc.) 

2. Acquire and install computer resources (hardware and 
software ) . 
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3. Train program content experts and instructional designers 
as needed. 

4. Assist in design and evaluation. 

5. Acquire, install and monitor telecommunications equip- 
ment . 

6. Select appropriate delivery media. 

7. Supervise input and revision of test items, 

8. Monitor system (hardware and software) performance. 

Management 

1. Approve project plan, evaluation plan; allocate resources . 

2. Assign staff. 

3. Review results, costs, resource requirements. 

4. Consider implications . 

5. Plan for installation and maintenance of the system. 

Paraprof essionals 

1. Supervise CML lab. 

2. Input new items, revise existing items, as directed. 
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3. Print and distribute records. 

4. Print and forward student notes requiring instructor 
attention . 

5. Input materials/ as directed. 

6. Perform other maintenance and clerical tasks as required. 

7. Post, monitor and manage terminal schedule. 

8. Assist with orientation of new students. 

Components of the CML Environment 

Attachment C contains a listing of personnel, hardware, 
furnishings, software, courseware, telecommunications and train- 
ing/orientation requirements of a fully-developed CML application. 
Due to the early termination of the research phase of this study 
further evaluation of these components was not possible. 

Table 5 shows a comparison of costs of PLATO-based CML and two 
other forms of delivery of the Nursing Assistant Program, one in 
which posttests were manually administered, scored and results 
recorded ( "paper-and-pencil without scanner"), and another in 
which an optical scanner was used to score and record results. 
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Tabla 5: Costs of PLATO-basad OIL and Othar DaUvary nodes , Nursing Assistant Program, AUC Edmonton 



DalivBriJ Cnmnnnfln^ 


Monthly 
uiaryBs* 




Program 
SnarB* 




Paper and 
Pencil, with 
scanner 




Paper and 
Pencil, without 
scanner 


MuL UllJQl a 


«7gg nn 

S/DD i uu 


iNOtB U 


5305,00 




nil 




nil 




Jul LUlal a 


ni 1 

nil 




nil 




$150,00 


(Note 2) 


$50,00 




iBiBCOmmuni cat ions 


ca 

SlJc.bO 


(Note 3) 


$53,00 




nil 




nil 




Subscriptions 


si050.cn 


(Note 4) 


$450.00 












Staffing 

.Sunfirvisinn nF hnnhs 

• tJUplOI) V A 9J.UI 1 Uk LOS La) 

•Scanning, scoring and 
rBCordkeaninn 


3DCJ iDJ 

nil 

nil 


fNnfn CI 


rDfVf Oft 

nil 


Q NOta b) 


5355 ilo 

(El 4 Art 

£71 .00 


fit L_ *1 \ 

(Note 7) 
(Note 8) 


$965.16 
SIOOB.OO 


(Note 7) 
(Note 3) 


•jluudiil aiiuwuriLas 


nil 


INOCB 10 J 


nil 


(Note 11) 


5650.00 


(Note 11) 


SB50.00 


(Note 11) 


TOTAL MONTHLY COSTS (AUG) 


$2571.13 




S986.B0 




51B3B.1B 




52673.16 




MONTHLY AVERAGE PER STUDENT" 


$17,14 




5B.5B 




$12.24 




S17.B2 





•As thB Nursing Assistant program usbs PLATO CML 3 hours per day CavBrage) the amount 'charged' is three-eighths (405;) 
of the 'actual' amount, 

••Assumes ISO students per year, Includes ReFresher students (25 in 19B6-B7). 

NotB 1: Terminals at S2750 sach x 6, amortized over 5 yBBrs - $275/mantfi. Monthly maintenance at 550 per terminal, 

530 for modem and multiplexer - $390 pBr month, Extra file parts ■ 5100 pBr month. 
Note 2: Paper, printing, optical scanner ansniar nhaeta, 

NotB 3: Conditioned tBlaphonB linB (no charge: paid fey APUSS), Multiplexers (2) at 52960 each, ovBr 5 years - SB9.67 

per month, Modem at 52570, over 5 yaars - $42, B3 par month, 
NotB 4: Subscriptions at 5350 per month each x 3 - 51OS0. (While AUC pays for 3 subscriptions, access is provided to 6 

subscriptions on the understanding that usage maxima and minima will not exceed Fulltime usb of 3 subscriptions. 
NotB 5: Instructional Assistant tine: $10,39 par hour x 60 hours per month (3 hours par day) - $623.63, 
NotB 6; Actual time charged is for 1 hour per day. 

Note 7: Instructor tima: 20.5 hours par month (41 tBsts CavBragal per month x 30 minutes pBr tBSt) x $47, 0B per 

instructor contact hour (B0 instructor contact hours pBr month, 10 months per instructor work gear, $37665 pBr 
instructor uork year C19B6 avBragB Program instructor salary]), 

Note B: Instructional Assistant, clerical time: 10 minutas pBr test, 41 tests [average] per month, $10.39 per hour 
■ $71.00, 

Note 9: Instructional Assistant time: B0 hours x $12,60 per hour CratB for former Instructional Assistant] - SIOOB.OO. 
Note 10: All CML testing currently dons outsidB of cIbss tima. 

Nota 11: 104 total tests pBr student at 30 jiinutBS CavaragB] per in-class test - 52 student class hours x 150 students 
(not including Refresher] x $1 per student class hour CaverBge sponsorship ratp, Nursing Assistant Program 
students, 1985-86] divided by 12, 
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The following points may be made based on the data in Table 5: 

1. While the use of the optical scanner materially lowered 
the average monthly testing and record keeping cost per 
student (from $17.82 to $12.24 per month), it did not 
produce conditions permitting student self-direction or 
self-pacing . 

2. In paper-and-pencil testing, both with and without use of 
the optical scanner, approximately 10% (52 hrs.) of the 
students' total in-class time was given over to test 
administration, marking and discussion. In PLATO-based 
CML all of this time is recovered, while total testing 
time (all of which occurs outside of class time) is 
reduced, perhaps as much as 60% (exact figures will be 
available in May, 1987). 

3. CML permits student sel f -d ir ec tion and self-pacing. 
Students in fact do take advantage of this capability (see 
Tables 3 and 4 above ) f and voice approval of it (see 
above, students 1 attitudes and practices). 

4. Presently, PLATO resources are used approximately 3 hours 
per day, leaving 5 hours per day available (not including 
evenings) . The Transitional-Vocational program uses 
approximately 2 hours per day, for 5 months of the year. 
Thus there is unused PLATO capacity for other programs, 
and such use would reduce the per capita and per hour 
costs of the system. 

31 
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5. Presently, $207.80 per month is expended for Nursing 
Assistant Program test supervision, consisting of 1 hour 
per day spent by the Instructional Assistant in the CML 
lab. (Supervision by instructors, as it is not considered 
"contact time," is not charged). It may not be necessary 
to supervise the lab constantly - the School of Nursing, 
the University of Alberta, provides 100% unsupervised 
PLATO CML time to its students (see Recommendation 3.4, 
below) . 

Conclusions and Recommendations 

Conclusion 1: All but one recommendation of the Montgomerie 

report (1985) were met fully and one recommendation 
met partially in this PLATO-based CML implementa- 
tion. The recommendations fully met were: 

1.1 CBL was incorporated in the Nursing Assistant 
Program. 

1.2 CML was incorporated before CAL. 

1.3 A team approach to development and implement- 
ation was used. 

1.4 A full-time CBL Coordinator was appointed. 
( Montgomerie 1 s recommendation was that a 
half-time coordinator be appointed. For most of 
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this p oject, a full-time coordinator was 
available . ) 

1 . 5 Computing facilities were contracted from 
another institv. .on (the University of 
Alberta ) . 

1.6 The PLATO CBL system was used. 

1.7 No attempt was made to integrate CML records 
with the Student Information System. 

One recommendation was not met: 

1.8 A centralized instructional design group was 
not established . 

Recommendation 1.1 A centralized instructional design unit 

with responsibility and capabilities for 
project development should be estab- 
lished. 

Conclusion 2: The evaluation process in place from December/ 

1985, to November, 1986, was successful. In 
November, 1986, changes to the project team and to 
research conditions initiated by the Nursing 
Assistant Program resulted in limitations of 
student access to CML resources and disrupted 
communication patterns among CML team members . 
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These developments forced premature termination of 
the research phase of the project. 

Recommendation 2.1: An instructional development team, as 

described by Montgomerie, should be used 
in CML project development , compr is ing 
content, instructional design, and 
computing expertise. 

Recommendation 2.2: During i mplement a t i on of research 

projects by the Division of Research and 
Evaluation, involved program team members 
should be seconded to the Division. 

Recommendation 2.3: During the research phase of projects, 

development and implementation processes 
should be the coordinated by the Division 
of Research and Evaluation. 



Conclusion 3: 



Training and scheduling of five regular and two 
Refresher classes of Nursing Assistant Program 
students was effectively and efficiently performed. 
Experience showed that this number of students 
(approximately 175) could be accommodated as 
jcheduled within existing resources. 
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Recommendat ion 3.1: 



Recommendat ion 3.2: 



Recommendation 3 . 3 



Recommendation 3.4: 



Recommendation 3. 5: 



Recommendation 3. 6: 
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All regular and on-site Refresher Nursing 
Assistant classes should use PLATO CML 
henceforth . 

The present student orientation materials 
and program should be used. However, the 
optimum amount of time and resources for 
student training should be assessed 
further and the orientation process 
revised. (Student use of notes, "term- 
comments," and various records [Grade 
Book] should be investigated.) 

Optimum terminal time requirements should 
be assessed to assure maximum efficiency 
in allocation of resources, balanced by 
respect for students 1 preferences . 

Use of unsupervised CML time should be 
investigated. 

The possibility of providing more "prime" 
testing time (between 9: 00 a.m. and 3: 00 
p.m. ) should be investigated, especially 
for students "unaible to access early and 
late times. 

Use of time saved by CML (approximately 
50 hours of class time formerly used for 
in-class testing) should be described. 
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Conclusion 4: Staff training achieved the goc.l of making instr ac- 
tors comfortable with CML, ana capable of using 
Instructor utilities. 

Recommendation 4.1: Present staff training components (manual 

and on-line or ientat ion) should be 
maintained. The usefulness and effec- 
tiveness of various components should 
continue to be assessed and the process 
revised as required. 

Conclusion 5: The present student schedule (7:30 - 8:30 a.m., 

12:00 - 1:00 p.m., 2:30 - 4:30 p.m.; total 4 hours 
per day) is adequate; however, in the opinion of 
some students it should be supplemented by evening 
availability. 

Recommendation 5.1: PLATO terminals (perhaps located in the 

LRC) should be available to Nursing 
Assistant Program students after 4:30 
p »m. 



Conclusion 6: Present posttests were suitable for present purpos- 
es . However, only two-thirds of available PLATO 
file space was used by the 1863 questions in the 
present item bank. 
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Selective item analysis should be done on 
present items and revisions made as 
necessary. 

New posttest items should be written to 
supplement the item bank, until the bank 
is approximately 90% f ull . (Approxi- 
mately 1000 additional items could be 
accommodated in the bank. Modules which 
item analysis show would benefit most 
from the additional items should receive 
attention first . ) 

Conclusion 7: Students were quickly capable of using the system 

effectively, and reported finding PLATO a useful 
and enjoyable experience . 

Recommendation 7.1: Other possible users of the CML materials 

should be identified, especially where 
flexible delivery/access are desired, and 
where students present a range of charac- 
teristics and motivations . 

Recommendation 7.2: Where off-site users are identified, 

pilot projects should be developed, as in 
this study, to test the utility and 
appropriateness of the present PLATO CML 
system. 



Recommendation 6. 1: 



Recommendation 6. 2: 
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Conclusion 8: Incorporation of CML resulted in actual cost 

savings for on-site delivery of this program. 

Recommendation 8. It The extent to which costs could be 

reduced further (i.e., by use of unsuper- 
vised time; see Recommendation 3.4) 
should be investigated. 

Recommendation 8.2: Potential cost reductions for off-site 

implementations should be investigated* 

Conclusion 9: The results of the CML project support off-site use 

of these materials and procedures, with adjustments 
and modifications for local conditions. 

Recommendation 9.1: Elements of the CML environment not 

included in this implementation and 
appropriate to off-site needs should be 
incorporated as soon as possible 
(Attachment Cj , 

Recommendation 9.2: Cost savings or greater efficiences with 

CML should continue to be sought, either 
by shortening of the program or by 
addition of components not now included. 

Recommendation 9.3: Advantages of student self -pacing and 

self-direction should be measured and, 
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where possible, quantified (i.e. in terms 
of student satisfaction, lower termina- 
tion rates , more self-assurance, appeal 
to a wider student clientele, better 
preparation, etc . ) 

Recommendation 9.4 Off-site implementation of CML should be 

systematically planned and implemented, 
to accrue the advantages identified in 
this study. 

Conclusion 10: A number of research questions were not addressed 

in this truncated evaluation. The following are 
questions which should be addressed: 

10.1 How do students use unsupervised CML time? 

10.2 What is the relationship between success 
indicators (level exam scores , clinical 
performance, etc.) and CML outcomes? 

10.3 What use could be made of on-line or scanner- 
based item analysis information in re vis ion 
of the existing test bank? 

10.4 Which CM j resources (including but not 
1 imited to those in Attachment C) are cost 
effective enhancements to the instructional 
program? 
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10. 5 What changes to staff roles occur as the CML 
package matures? 

10.6 What alternate technology (Micro PLM) could 
be used to supplement or replace on-line PLM, 
on-site and of f -site? 

10.7 What information is required in planning 
further use of CML, on-site and off-site? 
What research should be conducted to acquire 
this information? 
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Attachment A 



A-l 



Evaluation Plan 

Nursing Assistant Program Computer-Managed Learning Implementation 

September, 1986 
P. Fahy 

Director, Research and Evaluation 
July, 1986 



Goal 



To implement computer-managed learning (CML) via PLATO with « complete class of 
Nursing Assistants. 



. 1 . Train and schedule the September Nursing Assistant class in use of 
PLATO-based module posttests. 

2. Observe students in the CML testing environment and make adjustments to 
training, scheduling, and materials as needed. 

3. Determine the effect of the CML learning environment on the instructor^ 
role. 

4. Determine the effect of the CL learning environment on students' 
attitudes . 

4.1 Self-direction 

4.2 Motivation 

5. Determine the effect of the CML learning environment on t^me utilization 
for learning activities other than testing. 

5.1 Lab Practice 

5.2 Self-study 

5.3 Classroom 

5.4 Other media 

6. Determine the effect of on the Educational Assistant's role. 

6 . 1 Recordkeeping 

6.2 CML Lab Supervision 

7. Determine the time utilized in CML for various recordkeeping tasks. 

7.1 Archiving 

7.2 Weekly report generation 

7.3 Performance test results inputing 

8. Determine the effect of CML on the success rate of students on posttests 



Objectives 
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Methodology 

The following five methodologies will be used to gather data for evaluating the 
success of this project (See also Appendix A), 

1. Checklists, questionnaires, surveys, logs 

2. Observations and comparisons 

3. Analysis of test data (item-analysis) 

4. Interviews and discussions 

5. Voluntary recommendations of students, instructors, aides, and the manager 
Timeline 

The overall time-frame within which this evaluation will be conducted is 
September 2, 1986 - May 29, 1987 (the term during which the September Nursing 
Assistant class will be enrolled). The evaluation will consist of three levels 
corresponding with the three levels of the program for the September class: 

Level I - September 2-0ctober 10; 

Level II - October 13-January 23, 1987; 

Level III - January 26-May 29. 



Budget 

The primary expenses associated with this evaluation are the time required of 
the Division of Research and Evaluation and the CML Manager, Nursing Assistant 
Program. Research and Evaluation staff will reserve time as follows: 

Level I - 40% 

Level II - 20% 

Level III - 20% 
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Appendix A 



A-3 



Objective 

1 

S 

3 

4.1 
4.2 
5.1 
5.2 
5.3 
5.4 
B.l 
B.2 
7.1 
7.2 
7.3 
B 



Evaluation 
riethodCsD 
12 3 



! x 



x 
x 

X 
X 
X 
X 
X 
X 
X 
X 



X 
X 



X 
X 
X 
X 
X 
X 
X 



X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 



5 

X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
X 
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Terminal Requirements for CHL 



CLASS N Sept Oct Nov Dec Jan Feb "ar 

2« B 15 22 29 6 13* 20 2? 3 10- 17 54 1 Q IS 22" 29 5 12 19 2G 2 9 W 23 2 ^9 « 

"Itfl "l 2 3 "4 5 6 "1 "1 "0 11 12 13 14 IS 1G 17 IB 19 20 21 22 23 24 2S 2S 27 2B 

BEPT 1/BE 30 2 G 4 7 5 3 5 0 4 0 7 0 6 0 9 0 4 0 B 0 8 0 B 0 4 0 7 0 



Upfi „u 1 2 3 4 5 6 7 B 9 10 11 12 13 14 15 16 17 IB 19 20 21 

DC?20/S 6 30 2 5 4 7' 5 3 G 0 3 0 B 0 7 0 7 0 9 0 B 0 9 

> 

I 1 1 ) I 1 I 1 1 l l l s ie 0 13 7 1 ] J 

DECB/B6 30 2 S 4 7 5 3 6 0 3 0 B 0 7 0 ^ 

3 

hum 1 2 3 4 5 G 7 $ 

WBBk t) 3547535 5 

FEB 2/B7 30 ^ 

0 

Ueek It 

M 23/B7 30 

Sessions: 1 60 180,120 210 150 90 150 60 270 120 420 150 270 1B0 330 240 240 0 450 390 300 <K0 270 420 390 570 330 510 450 
Hours of 

Eachto, 0.6 1,5 11,9 1.3 0,8 1,6 0,5 2,3 1 4,4 1,3 2.3 1,5 2.B 2 2 JJ.B ^J^^^^^J^i^j;?. 

•4-day wsBk. 
•♦2-day weak. 
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CP1L Checklist 

1 . Personnel 



1.1 Instructors Complete 

1.1.1 Describe role, duties of participants 

1.1.2 Develop specific orientation modules For 
participants 

1.1.3 Arrange information sessions For staff ZZZ^ZZ 

1.1.4 Keep log of activities 

1.1.5 Evaluate 

1.1. 6 Revise 

1.1.7 Document 

1.2 AideCs) 

1.2.1 Describe role, duties 

1.2.2 Plan and conduct orientation 

1.2.3 Plan and conduct detailed training ZZZZZZZ 

1.2.4 Establish reporting/supervision 

1.2.5 Evaluate 

1.2.B Upgrade training as needed 

1.2.7 Document " "~ ~~ 

1.3 Uolunteers CPeer tutors) 

1.3.1 Describe role 

1.3.2 Recruit '~ 

1.3.3 Train 

1.3.4 Establish supervision 

1.3.5 Evaluate 

1.3 .6 Document 

1.4 Admir -trators, managers 

M , 1 Describe role 

...2 Orient 

1.4.3 Evaluate 

1.4.4 Upgrade, as requested 

1.4.5 Document 

1 .5 Evaluators 

1.5.1 Describe role 

1.5.2 Select Cinternal/externalD _ 

1.5.3 Orient 

1.5.4 Evaluate 

1.5.5 Document ~ 

1.6 Students 

1.6.1 Develop orientation modules 

1.6.2 Present orientation 

1.6.3 Evaluate 

1.6.4 Revise 

1.6.5 Document 

2 • Courseware 

2.1 On-line 

2.1.1 Input content 

2.1.2 Proof content [^^^ 
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2.1.3 Pilot 



2.1.4 Evaluate 

2.1.5 Revise 

2 . 1 .6 Document 

2.1.7 Install 

2.2 Off-line materials, activities 

2.2. 1 Identify sources 

2.2.2 Acquire 

2.2.3 Assure accessibility 

2.2.4 PilDt 

2.2.5 Evaluate 

2.2.6 RevisB/replace 

2.2.7 Document 
2.2.B Install 



2.3 Media 



3. Environment 

3.1 Room 



2.3.1 Select 

2.3 .2 Acquire 

2.3.3 Pilot 

2.3.4 Evaluate 

2.3.5 Revise/replace 
2 . 3 . G Document 
2.3.7 Install 



3.1.1 Renovate 

3.1.2 Wire 

3.1.3 Furnish 

3.1.4 Evaluate 

3.1.5 Renovate further 

3.1 .6 Document 

3.2 Furnishings 

3.2.1 Identify 

3.2.2 Acquire and arrange 

3.2.3 Evaluate 

3.2.4 Replace/rearrange 

3.2.5 Document 

3.3 Schedule 

3.3.1 Calculate terminal timB needed 

3.3.2 Identify terminal locations 

3.3.3 Establish schedule posting locationCsD 

3.3.4 Evaluate schedulB utility 

3.3.5 Revise schedule and procedures 
3.3.G Document 

3.4 Study materials access 

3.4.1 Provide storage with access/security 

3.4.2 Evaluate 

3.4.3 RearrangB/change procedures 

3.4.4 Document 

4. Communications 

4.1 Records 

4.1.1 Identify needed data/records 

4.1.2 Establish data access 

4.1.3 Establish data archive 

4.1.4 Document a q 
O 4.2 Reports 

ERIC 



4.S.1 Identify neBded reports 

4.5.5 Establish reporting timetable • 

4.5.3 Establish report formatCs) 

4.5.4 Establish report recipients 

4.5.5 Document 
4 .3 Telecommunications 

4.3.1 Establish communications network 
4.3.5 Orient users 

4.3.3 EvaluatB utility 

4.3.4 Document 

5 . Hardware 

5.1 Terminals 

5.1.1 Identify delivery terminal CsD 

5.1.5 Develop user orientation 
5.1.3 Monitor reliability/utility 

5.1.3 Evaluate 

5.1.4 Revise orientation/replace terminals 

5.1.5 Document 
5.5 Modems/multiplexers 

5.5.1 Acquire 
5.5.5 Install/test 

5.5.3 Monitor 

5.5.4 Evaluate 

5.5.5 Replace 

5.5.6 Document 

5.3 Printers 

5.3.1 Acquire 
5.3.5 Monitor 

5.3.3 Evaluate 

5.3.4 Replace 

5.3.5 Document 

5.4 Monitors 

5.4.1 Acquire 
5.4.5 Monitor 

5.4.3 Evaluate 

5.4.4 Replace 

5.4.5 Dnnumpnt 
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ATTACHMENT D 

D-l 



ORIENTATION TO C.M.L. (PLATO) 
LEARNER'S MODULE 

THIS IS WHAT YOU WILL LEARN: THIS IS HOW YOU WILL LEARN IT: 



I Define: 


I Terminology: 


a) 


C .M. L. 


a) 


C.M.L. : Computer Managed 


0) 


pi ATrt fD T M "\ 
srLkALKJ \r»lj •£!>•) 




Learning a system that uses 


c) 


Terminal 




computers to issue and mark 


d) 


Rewrite 




post tests, then keep records 


e) 


Remediation 




of student progress through the 


f) 


Self-pacing 




tests in each course. * 






b) 


PLATO (P.L.M) Learning Manage- 








ment; a computer system leased 








to AVC. from U of A. 






c) 


Terminal: the screen and key- 








board used to communicate with 








the main computer at U of A. 






d) 


Rewrite: post- test 8 in a 








module are rewritten until 80% 








mastery is achieved. 






e) 


Remediation: acquiring extra l 








help with the material in a 








module when not successful 








after 1 original and 2 








rewrites . 



f) Self-pacing: Students in Level 
II & III can accelerate or 
decelerate to a degree, depend- 
ing on individual need and 
clinical facility available. 



Students will choose terminal 
time that will suit their 
Individual readiness to write 
post-tests. 
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THIS IS WHAT YOU WILL 


LEARN: 




THIS 


IS HOW YOU WILL LEARN IT: 


II .Discuss the advantages 


of C.M.L: 


a) 


In General: 


a) In general 






i) 


Reduces time for Clerical 


b) For Instructors 








duties • 


cj ror students 
















ii) 


Improves Record-Keeping 










Q\r of om 
Oj BLcUlt 








ill) 


Gives more accurate data for 










cValUallUU UJ, LcoL ILculo • 








iv) 


Provides group assessment of 










post l68l psnonnanCc • * 








v) 


Provides information regard- 










4 no o ^ii /I t>n t" « oi"f fsy-man^ o 
lUg oLUQCllL pcil ULlildnUc • 








vi) 


Supplies random selection of 










focf" A ♦- omo t^v t"li o O Ottitm * t" ot" 
LCOL ILCUlO UY LllC CUUUULCi • 








v) 


. Xnceases protability outside 










of AVC Edmonton, 








vi) 


Increases flexible entry and 










exit. 








vii) 


Increases self-pacing/self-[ 










directed components* 








viii) 


Manages student testing and 










record-keeping 






b) 


For Instructor 








i) 


Brief orientation required. 








ii) 


Frees up Instructional time 










from test marking, for indi- 










vidual student instruction. 








iii) 


Provides reliable feedback 










on individual student 










progress • 








iv) 


Can handle large numbers of 










students per terminal out- 










side class time. 
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THIS IS WHAT YOU WILL LEARN: 



THIS IS HOW YOU WILL LEARN IT: 



v) Allows for communication 
between instructors and 
students via the 
"message/note" facility, 

vi) Will allow more time for 
practical skills practise 
and assistance with 
problems • 

vii) Teacher as facilitator, 

c) For Students: * 

v i) Require only brief orienta- 
tion to terminal. 

ii) More independent student 
choice. 

iii) Time conservation for 
students : 

- some self-pacing in a 
week. 

- may choose own times to 
write. 

- no waiting for classmatesE 
as in In-class writing. 

- No marking in class. 

- Immediate feed-back: 
(correct or incorrect and 

objective to study if 
incorrect) • 

iv) Increases motivation to be 
successful at terminal due 
to immediate feedback. 

v) Allows for communication 
with Instructor via 
"message/ note" system. 
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THIS IS WHAT YOU WILL LEARN: 


THIS HOW YOU WILL LEARN IT 


III) Be 


orientated to the PLATO 


■ — p — ■ 

a) Scheduling: 


terminal: 




a) 




- Master sign-up sheet in Room 604. 


Scheduling - appt. cards 


- The student can sign up for one 


b) 


Sign-on to terminal 


or two 15-minute time slots at a 


c) 


Terminal use 


time for next day. Not 


d) 


Choices 


consecutive. 


e) 


Lock out - Remediation 


- Extra time may be be signed up 


f) 


Post-test discussion 


for up for as the need arises. 


g) 


Attendance 


- If less time is needed the 


h) 


Review of student records 


student will remove his/her name 


i) 


Messages /Notes 


from the schedule so someone else 


j) 


Test - stop options 


may use that 15-minute time slot. 


k) 


List of key used 


- As many post-tests as the student 






is able to do may be done per 15- 






minute time-slot. 






- Terminal sign-up is on a first- 






come first-serve basis. 






- Consistant study habits are a 






must in order to make scheduling 






decisions • 






- All post-testing including 






rewrite must be complete by 






Friday afternoon otherwise clini- 






cal competencies will not be able 






to be done the next week. [ 






- (See appt. Cards) These 






reminders of times signed-up for, 






may be filled in at the sign-up 






time. 






- If a student forgets his times , 






someone else could use these 






slots. 






- *Changes must always be made on 






the master sign-up sheet. This 






is Durelv student rpflnonaibl H 






to get enough time to complete 






all post-tests. 






d j oign on i erminax • v«e e screen 






displays.) Refer to Addendum. 






- Type your PLATO name. Last 






name; first initial (no capitals 






necessary) • 



I 
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THIS IS WHAT YOU WILL LEARN: 


THIS IS HOW YOU WILL LEARN IT: 




- Type your PLATO' grAip. Na,l, or 
2 or 3 etc. 

- See Next Key , during hands Qn 

- See Shift Stop Key. J orientation 

- Type your password (5 or more 
digits; letters or numbers). 



every 60 days. 
- Sign-off-shift/stop 

c) Terminal Use: 



ALWAYS READ ENTIRE SCREEN BEFORE 
CHOOSING AND PRESSING A KEY. ' 

- See TERM. ANS. Key - pressed 
after answer choice is made 

- You may change your answer ONLY 
BEFORE you press the ANS. key - 
once pressed this locks the 
answer into the computer for 
processing. 

- CORRECT OR INCORRECT is then 
your feedback. If incorrect you 
will be told which objective to 
study. 

The question code is in the top 
right corner of the screen. - r 
i.e. Na 86b l.lg | 

+ * 
Nurs. Assist Mod. 



Question no. 



L 



Test 
Version 



objective 
No. 



- Write this code iown if you wish 
to discuss the question later. 

d) Choices : 

i) Skip option : You may skip a 
test question and it will 
reappear at the end of the 
test for you to answer. 



55 

5 

ERIC 



D-6 



THIS IS WHAT YOU WILL LEARN: 



THIS IS HOW YOU WILL LEARN IT: 



ii) If you run out of time while 
doing a test you will sign 
off [Shift-stop] and resume 
fcb/e test at that point when 
you next sign on. You must 
always sign-off when having 
the terminal so a new 
student can sign on. 

e) Lock-out Remediation : 

- You will be allowed to do 3 
post-tests on the same module- i 
If still unsuccessful the 
computer will lock you out of 
this module for 1 hour which 
allows time for ycu to see your 
Instructor to make an appoint- 
ment for assistance. Then you 
can continue testing, on the 
next module immediately. 

f ) Post-Test Discussions : 

- You may see your Instructor 
regarding a problem question or 
in a group if several students £ 
are having similar problems. 

- Remember to record the question 
code if you're having problems 
so your Instructor can refer 
back to the question. 

g) Attendance : 

i) In Class is compulsory even 
if you've already passed the 
post-test. 

ii) Lab attendance is compulsory 
for demonstration, practise 
and performance testing. 

iii) Clinical attendance 



7 Acute Care weeks unless 
accelerating or decelerat- 
ing. (This would be 
explained i i. an individual 
basis) . 



7 Extended Care weeks 
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THIS IS WHAT YOU WILL LEARN: 



THIS IS HOW YOU WILL LEARN IT: 



P 

h) Review of Student Records (See 
examples) Refer to Addendum. 

- Module Index 

- "Nursing Assistant Module Post- 
tests." 

i) Messages/Notes 

i) From Students to 
Instructors ; 

- if having difficulty a 
student could send a * 
message on the terminal to 
the instructor. 

- See Screen Displays in 
Addendum. 

- Or: While being tested 
press -Shift /Term, and 
type comment* Then type 
your message 

ii) From Instructors to 
Students: 

- a message from the 



j) Teat-Stop Option : 

- a test will be stopped early if: 
i) you have already 



Instructor could appear on 
the screen immediately I 
after you sign-on, 
ALWAYS READ THE ENTIRE 
SCREEN CAREFULLY. 



incorrectly answered the 
allowed "quota" per post- 
test 



ii) 



you have correctly answered 
the required number of 
questions for that module. 



k) List of Keys Used : 



Next 
Shift 



Erase 
Lab 

Term Answer 



Stop 
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ORIENTATION TO C.M.L. (PLATO) 



INSTRUCTORS MODULE 



THIS IS WHAT YOU WT.LL LEARN: 



THIS IS HOW YOU WILL LEARN IT: 



Define: 



I Terminology : 



b) 
c) 
d) 
e) 
f) 



C.M.L. (Computer Managed 

Learning) 

PLATO (PLM) 

Terminal 

Rewri te 

Remediation 

Self-pacing 



a) C.M.L. : A system that uses 
computers to issue and mark 
post-tests, then keep student 
records of their progress 
through the tests in each 
course. 



b) (P.L.M.) Plato - Learning 
Management : is a computer 
system leased to educational 
institutions in northern 
Alberta from U of A. 



c) Terminal : is made up of a 
screen and keyboard that allows 
information to be communicated 
from the main computer at U of 
A to the students/Instructors 
at AVC. ? 

d) Rewrite : in competency-based 
learning the student rewrites 

post-tests until mastery is 
achieved. 

e) Remediation : in the N.A.P. the 
students will be allowed to 
rewrite a post- test on one 
module x 3 at the terminal 
before tKEy are expected to ask 
for assistance from their 
Instructors. Seeking extra 
help is called "Remediation". 

f) Self-pacing : as adult learners 
in a competency-based program 
there is some provision for 
working ahead in modules. 
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THIS IS !JHAT YOU WILL LEARN: 


THIS IS HOW YOU WILL LEARN IT: 




f) Students in Level II and III 
may be able to accelerate or 
decelerate depending on indi- 
vidual need and clinical 
facility availability. 
- Students will be, in the 
school weeks, allowed to 
choose terminal time that 
will suite their individual 
readiness to write post- 
tests. 


(Teacher Only) 
II Understand the Purpose of CLM in 


II Background - History: 


the Nursing Assistant Program 


In Sept/79 the Aide-Orderly 
programs were amalgamated. A new 
learning system (Modular Compe- 
tency-Based) was designed and 
implemented. This is a student 
self-directed system. 

In July/85 after re-examination of 
the goals of this new learning 
system, it was decided that some - r 
were met in a limited manner and 1 
some not at all. Therefore a 
feasibility study into the use of 

CML was begun. The goals that 
still needed to and could be net 
with CML were: 

1. Portability to areas outside 
metropolitan areas. 

2. Individualized instruction. 

3. Flexible entry and exit. 

4. Provisions in environments for 
self-pacing and self- 
direction. 
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THIS IS WHAT YOU WILL LEARN: 



D-20 

THIS IS HOW YOU WILL LEARN IT: 



The study suggested fhat CML 
operating and managing student 
record-keeping and testing would 
save a ot of Instructional time 
and effort, plus facilitate 
meeting the aforementioned goals. 



Ill Discuss the Advantages of CML 

a) In General 

b) For Instructors 

c) For Students 



a) In General : 

CML will: 

i) Reduce time necessary for i 
routine clerical duties* 

ii) Improve record-keeping 
systems. 

iii) Give greater accessibility 
to more accurate data for 
evaluation and analysis of 
tests. 

iv) Provide for individual and 
group assessment and evalu- 
ation of post-test 
performance. r 

v) Supply support for efficient 
decision-making regarding 
student progress. 

vi) Supply random test item 

selection by the computer/ 
module. 

b) For Instructors : 

i) Requires only brief 

orientation to the system. 

ii) Frees instructional time 
from test administration, 
marking and tedious 
discussions. 
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THIS IS WHAT YOU WILL LEARN: 



THIS IS HOW YOU WILL LEARN IT: 



- no hand-recording 

- 1 x weekly check of 
individual student and 
general class progress. 

iii) Provides reliable feedback 

- Easy access to compre- 
hensive data on a student, 
if required. 

iv) Can handle a large number of 
students/terminal - outside 
of class time. 1 

v) Allows for constant communi- 
cation between Instructors 
and Students via the 
"message" or "note" 
facility, 

vi) Will allow more time for 
practical skills practise 
and individual help for 
students having 
difficulties. 

vii) Teacher role - "f acilitatorl* 
rather than "information 
giver" 

c) For Students : 

i) Requires only brief orienta- 
tion. 

ii) Androgogy: more individual, 
independent student choice. 

iii) Time conservation per 
student: 

- No waiting for classmates 

- less distractions when 
writing 

- No marking test in class 
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THIS IS WHAT YOU WILL LEARN: 



THIS IS HOW YOU WILL LEARN IT: 



iv) Immediate feedback 

- correct or incorrect 

- Refers to objective in 
mod* to study if answer 
incorrect* 



v) Increased motivation to do 
well at the terminals! 
students receive immediate 
feedback and reinforcement* 

vi) Allows for constant communi- 
cation between instructor 
and students via "message" 
or "note" facility 



Teacher Objective Only 

IV Discuss Program Changes with 
C .M.L . 

a) Immediate 

b) Projected 



a) Immediate : 

i) The utilization of 

post-tests as criteria for 
progression in program and 
for determining need for 
referrals to Learning 
Specialists remains the 
same. 



ii) More student control over 
testing situation* 

iii) More emphasis on student 
self-direction. 



iv) Record-keeping no longer a 
time consuming, tedious 
activity. Once-weekly 
activity to generate 
individual and group 
progress records* More 
often, only as necessary, 
for individual students* 
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THIS IS WHAT YOU WILL LEARN: 



THIS IS HOW YOU WILL LEARN IT: 



v) Encourages a re-evaluation 
of criteria used for 
Northlands award 
determination. 

vi) Encourages a re-evaluation 
of the records being kept to 
date and their necessity or 
expendability. 

vii) There will be more class and 
lab time for practice and 
perhaps even peer tutoring; 

viii) More emphasis on "practical" 
component of the program. 

ix) More thorough testing of 
objectives in modules. 

b) Projected : 

i) Post-tests could become 
optional, used only as 
review or study guides. 
Only major exams and 
practical skills would then!" 
be criteria for progression 7 
through program. 

ii) Could increase flexibility 
of the academic portion of 
the program, i.e. separate 
schedules for the acceler- 
ated, the average and the 
decelerated students 

iii) CML area for Health Careers 
only where format and 
utilization of environment 
would be optimal for student 
independent learning. 

iv) Student self-direction could 
include peer-tutoring. 
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THIS IS WHAT YOU WILL LEARN: 


THIS IS HOW YOU WILL LEARN IT: 




v) More emphasis Sa "practical 
skills" of the program* 
?More clinical and less 
class time, in Levels II and 
III. 

vi) Record-keeping will include 
performance data input on 
the SPSS System as well as 
entry of the Clinical 
Competency file at the 
completion of each class* 

vii) Northlands award criteria 
more heavily weighted in the 
clinical performance and 
participation areas. 

viii) Attendance policy: All 
students would sign-in th^n 
make a choice from several 
structured situations; as to 
which to attend, i.e. class, 
AV media viewing, self- 
study, practice. Note: All 
would attend demonstrations 
and Lab performance t 
situations. ? 


Student and Teacher 
V Be oriented to the PLATO terminal 


a) Scheduling: 


a) Scheduling - appt. cards 

b) Sign-on Terminal 

c) Terminal use 

d) Choices 

e) Lock-out-Remediation 

f) Post-test discussion 

g) Attendance 

h) Review of student records 
(one set for Instructors and 
one for students) 

i) Messages 

j) Test-stop options 
k) List of keys used 


— There will be a master sign— up 
schedule posted in the computer 
room 604. 

- The students may sign-up for two 
15-minutes time slots at a time 
(i.e. one for present post-test 
and one for the next day). If 
more time is needed for rewrites 
during the week these are signed 
for as the need arises. If less 
time is needed the students are 
asked to remove their names from 
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THIS IS WHAT YOU WILL LEARN: 



THIS IS HOW YOU WILL LEARN IT: 



the schedule. The students may 
do as many post-tests in one 
time slot sequentially as they 
are able, i.e. the average 10 
item post-test takes 5 minutes 
of computer time to do. 
Students may sign-up for only 
one (1) cime-slot at a time if 
they choose. However, terminal 
sign-up is on a first come, 
first: serve basis. 

- If students prefer morning or i 
afternoon writing times this 
must be considered when signing 
up. 

- Consistant planned study habits 
are a "must" in ordr - to make 
scheduling decisioi 

- All post-tests for t* / ,"n week 
must be completed or 
successfully by Friday 
afternoon, otherwise competency 
experience will be lost in the 
following clinical week. * 

- Ap; cement note pads will be 
av.! ;.*t>Ie for each student (see 
cop/) if they want; to fill in 
and keep, to remind them of 
their terminal times. 

- Terminal appointment times must 
be adhered to or they will be 
forfeited to the student 
signed-up after them. They will 
also have to find another time 
slot in which to complete 
post-testing. *The Instructors 
have ao control over this 
scheduling - it is purely 
student responsibility to ma T :e 
and keep their appointments. 



THIS IS WHAT YOU WILL LEARN: 
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THIS IS HOW YOU WILL LEARN IT: 



b) Sign-on Terminal : 

- Student types his PLATO name . 
The student name will be 
rostered Into the computer; last 
name then firet Initial (no 
capitals necessary). (Show 
screen displays.) The 
Instructor will type Instr. 

- Student types his PLATO group. 
The student groups will be Na^, 
Na 2 » Na 3 , Na^ Na 5 . (Describe; 
NEXT key, Shift STOP Key) 

- Student types his password 
(password will be individual 
preference, 5 or more digits; 
could be numbers or letters). 
(The Instructor's will be 
changed from time to time. 
PLATO prefers everyone's pass- 
word to be changed every 60 
days. 

c) Terminal Use : 

NOTE: ALWAYS READ ENTIRE SCREEN * 
BEFORE MAKING CHOICE, OR CHOOSING 
APPROPRIATE KEY. 

i) TERM ANS Key is pressed once an 
answer is made. This locks the 
answer into the computer and 
allows th^ computer to process 
it. Immediate feedback is given 
(correct, incorrect - if 
incorrect the objective in the 
the module to review is 
indicated). 

ii) The ffJESTION CODE will be in the 
top right corner thescreen, 
i.e. Na 86b 1.1 g 
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THIS IS WHAT YOU WILL LEARN : 


THIS IS HOW YOU WILL LEARN IT: 




«» 

Nurs. Assls. 

Question 
4 + 




Na 86b 1.1 ^4- version 
t ♦ 
Module Numbc r 1 

objective 




This code needs to be 
written down by the student 
in order to refer back to 
this question for discussion 
purposes with the 1 
Instructor (ie) any type of 
problem. 




iii) Sign off-shift stop 




d) Choices: 
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i) Skip-option - a question may 
be "skipped" during the 
course of the test but it 
will reappear at the end and 
will then need to be 
answered. 

r 

ii) If the 15-minute time period 
is up and t:he student is not 
finished th?. reft, he/she 
will "sign-off [shift STOP] 
and the next time the 
student signs on the test 
will resume at that same 
question. 

Students must always sign- 
off when leaving the 
terminal so a new student 
can sign-on. 

e) Lock-out: REMEDIATION 

- a student will be allowed to 
write 3 randomly selected post- 
tests per module, then will be 
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THIS IS WHAT YOU WILL LEARN: 



THIS IS HOW YOU WILL LEARN IT: 



instructed by the ffbmputer to 
seek assistance from their 
Instructor. The student is not 
allowed access to any more post- 
test attempts until this "lock" 
is removed - 1 hour later which 
gives the student the time to 
make a remediation appointmeat 
time. They can sign up for 
another module post-cest 
immediately. 

- Automatic record-keeping 
indicates: . 

i) How much terminal time the 
student has used* 

ii) How often each post-test 
was attempted. 

iii) Success or incompleteness 
of each module 

f) Post-Test D iscussions : 

- will v * -it the discretion of the 
Instructors on a team, as to r 
format . * : 

- Discussions could be group >; l 9 
2, etc. /week (scheduled) or 
on an individual basis. 

- Students must record question 
epce in order to assist the 
Instructors to find the question 
for discussion. (Refer to part 
part (c) Terminal Use (iv) for 
code description. 

8 ) Attendance : 

i) Classroom is compulsory even 
if a student has passed a 
post-test on the module 
being presently discussed* 
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THIS IS WHAT YOU WILL L3ARN: 



THIS IS HOW YOU WILL LEARN IT: 



Attendance is called in 

the morning and afternoon as 

usual. 

ii) Lab attendance is compulsory 
for all students for demos, 
practise and performance 
testing. 

iii) Clinical attendance require- 
ments remain the same, to 
fulfill the practical skills 
portion of the program. 

h) Review of Student R ecords ; 

i) For Instructors ? 

- "Nursing Assistant Module 
Post-tests" (See 
overheads. ) 

- Curriculum Grp Records, 
i.e. neme, class ABCDEFFG 
(See overheads.) 

- average Test Duration 
(See overheads.) 

- Individual Student Record, 
l«e« Mod. Status, Mastered 
Score, Test (See 
overheads • ) 

ii) For Student 

- "Nursing Assistant Module 
Post-tests" (See 
overheads.) 
Group records will be 
generated once a week, 
i.e. on Thursday afternoon 
so the Tean Leader knows 
which students need to 
"catch up" on Friday and 
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THIS IS WHAT YOU WILL LEARN: 



TTHIS IS HOW YOU WILL LEARN IT: 



who will not be able to do 
competencies clinically 
the following week. 



i) Messages: How to Send : 

i) From Students to Instructor 
(See overheads .) 

ii) From Instructor to Student 
(See overheads .) 



j) Thorough-Testing Capability 

The computer has been programmed 
to select a specified number of 
questions randomly, ensuring all 
module objectives are thoroughly 
tested. 

Ear ly-Tes t-S top : 

- It is possible for a test to 
stop if the student has answered 
his/her "limit" of incorrect E 
questions. This 1) saves time, 
2) stops a student who needs 
more review, and 3) keeps the 
student from seeing the entire 
bank of questions. 

- This option can be used if a 
student has correctly answered 
the required number of questions 
as well. 

k) List of keys used : 

Next Erase 

Shift Lab 

Stop Term. Ans . 
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ATTACHMENT E — Pretest 

E~ 1 

COMPUTER ADAPTATION SCALE 

Name 



0 12 3 4 
0 12 3 4 



0 if you DON'T KNOU! or HAUE NO OPINION 
Circle: 1 if uycu STRONGLY DISAGREE 

2 if you DISAGREE 

3 if you AGREE 

4 if you STRDNGLY AGREE 

DK SD 0 P SA 

1. I think using a computer For uirxting tests would 

make me nervous • 0 12 3 4 

2. I think using a computer For writing tests would 

be easy . 0 12 3 4 

3. I think using a computer For writing tests would 
be slower than having them written on paper and 
marked by the teacher. 

4. I think using a computer For writing tests would 
be too impersonal . 

5. I think using a computer For writing tests would 
make me Feel too isolated. 

B. I think using a computer For writing tests would 
be more Flexible than paper tests. 

7. I think using a computer for writing tests would 
be more eFFicient use oF my time. 

B. I think using a computer For writing tests would 
be satisFying. 

9. I think using a computer t'jr writing tests would 
be Frustrating 

10. I think using a computer For writing tests would 

be interesting. 0 12 3 4 

11. I think using a computer For writing tests would 
improve my knowledge oF the subject matter more 

than regular tests. 0 12 3 4 

12. I think using a computer for writing tests would 

be enjoyable. 0 12 3 4 



0 
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Attachment F — Posttest 
COMPUTER ADAPTATION SCALE 



F-l 



Name 



Circle : 



0 if you DON'T KNOU) or HAUE NO OPINION 

1 if you STRONGLY DI SAGREE 

2 if you DISAGREE 

3 if you AGREE 

4 if you STRONGLY AGREE 

DK SD 



SA 



1. I found that using PLATO for writing .tests 
made me nervous . 

2. I found that using PLATO for writing tests 
was easy . 

3. I found that using PLATO for writing tests 

was slower than having them written on paper and 
marked by the teacher. 

4. I found that using PLATO for writing tests 
was too impersonal . 

5. I found that using PLATO for writing tests 
made me feel too isolated. 

6. I found that using PLATO for writing tests 
was more flexible than paper tests. 

7. I found that using PLATO for writing tests 
was more efficient use of my time. 

B. I found that using PLATO for writing tests 
was satisfying. 

S. I found that using PLATO for writing tests 
was frustrating 

10. I found that using PLATO far writing tasts 
was interesting. 

11. I round that using PLATO for writing tests 
improved my knowledge of the subject matter more 
than regular tests. 



12. I found that using PLATO for writing tests 
was enjoyable. 
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Attachment G 



G-l 



PRE-USE QUESTIONNAIRE RESULTS POST-USE QUESTIONNAIRE RESULTS 



CLASS START DATE TOTAL CLASS STAPT DATE TOTAL 



M THINK USING A COMPUTER 


OCT, 


e& 


DEC. 


Be 


FEB. 


B? 


N 


1 


M FOUND USING A COMPUTER 


SE p T. 


66 


OCT. 


66 


DEC. 


66 


FEB. 


V 


N 


1 


PDF TESTING 


















FDR TESTING --■ 
















N 


: 


N 


z 


N 


Z 






N 


I 


N 


1 


N 


1 


K 


* 
* 






- MOULD HAKE RE NERVOJS 


















- HADE HE NERVOUS 

























10 




i: 


l?: 


13 


2!Z 




56: 




10 


101 


16 


161 


16 


I?l 


20 


201 


66 


Dm <• 




1! 


le: 


7 


IIS 


9 


15i 


2? 


441 




6 


61 


7 


71 


9 




U 


111 


*r 


Tr«, 










— MAS EASY 
















4 


71 


j 


6Z 


1 


21 


10 


17: 




6 


61 


1 


i: 




61 


4 


4: 


J? 


1 7 1 




17 


26: 


14 


23: 


19 


32: 


50 


B31 




14 


141 


22 


221 




201 


27 


271 






- MOULD BE SLOWER THAN 


- MAS SLOWER THAN Pfl p ER 




87L 


rfrt^ fin? FcNmU 


















AN? PENCIL 
























**? 


231 




31Z 


23 




75 


931 




16 


161 


25 


251 


26 




30 


301 


97 


961 




1 


11 


2 


2Z 


3 


4Z 


6 


71 




2 


21 






1 


1: 


1 


1; 


4 




— MULt B c TOO lKPEP c r?KA 

t> JUL* v — i ww *nr *L™n* 










— MAS ?O0 IHPER C DNAL 

Pn* bW A|l» kn« JHnL 










41 




I! 


29: 


22 


3o: 


20 




63 


Bfcl 




i: 


161 


22 


231 




231 


2 C 


7" 


EE 








<: 


*> 


3Z 


5 


7: 


10 


u: 




3 


31 






3 


31 






6 


6: 


— MDL'LD MAKE HE FEEl top 








— HA nr HE FECI TOD 
















ISOLATED 


















ISOLATED 
























24 


30Z 


27 


34Z 


24 


30Z 


75 


951 




17 


171 


23 


231 


25 


251 


22 


re: 


93 


931 




2 


31 






2 


3Z 


4 


51 




1 


11 


2 


21 


2 


21 




:: 


i 


71 


- MOULD BE MORE FLEXIBLE 










-- MAS MORE FLEXIBLE 


















10 


13- 


7 


9Z 


10 


13Z 


2? 


351 


DISAEREE 


6 


6: 


r 




6 


61 


E 






261 
741 




18 


23Z 


19 


25Z 


13 


17Z 


50 


651 


AGREE 


1! 


in 


20 


:i: 


2»'» 


211 


2' 




?: 


- MOULD BE WORE EFFICIENT 


















- HAS HOPE EFFICIENT USE 






















USE 0 r TIME 


















If TIRE 
























3 


41 


2 


2Z 








61 






31 






1 


2: 


s 




D 


B', 




24 


291 


27 


33Z 


26 


32Z 


77 


941 




15 


151 


24 


24: 


15 


251 




27L 






- MOULD BE SATI£F\IN6 


- MAS SATISFYINB 








9! 






3 


41 


2 


3Z 




3Z 


7 


101 




4 


41 


t 


31 


4 


<: 


3 


3: 


14 


MS 




17 


25X 


**5 


371 


19 


261 


61 


9o: 




17 


17: 


20 


20: 


23 


231 


26 


26: 


66 


B6; 


— MOULD BE FRUSTRATING 




— MAS FRUSTRATING 


DISAGREE 


19 


261 


23 


32Z 


20 


:e: 


62 


661 




14 


151 


14 


15: 


20 


211 


26 


27: 


74 


77: 




4 


6Z 


2 


3Z 


4 


61 


10 


141 




4 


4: 


9 


91 


6 


61 


3 


31 


22 


231 


- MOULD BE INTERESTING 






- MAS INTERESTING 








2 


2Z 




:z 






4 


51 


DISAGREE 


1 


u 


1 


12 
















25 


30: 


27 


32: 


26 


331 


BO 


951 




16 


l?: 


23 


241 


24 


251 


30 








- MOULD IMPROVE HY 


~ IHPROVED HY KNOWLEDGE OF 








311 


95 


9b: 


KNON.EDGE OF THE SUBJECT 


















THE SUBJECT 
























E 


161 


B 


161 


B 


161 


24 


481 


DISAGREE 


6 


101 


9 


111 


4 


51 


B 


10: 


2° 


361 




C 




10 


20: 


B 


16: 


26 


521 




9 


n: 


6 


b: 


1? 




19 


241 


51 


6C 


MOULD BE ENJOYABLE 




- MAS ENJOYABLE 










211 








2 


3Z 


1 


1Z 






3 


41 


DISAGREE 


5 


s: 


1 


u 


2 


21 


1 


1: 


0 


91 




22 


31Z 


25 


35Z 


21 


30Z 


6B 


961 




13 


14: 


22 


231 


23 


241 


2P 


2 0 : 


66 


9i: 
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BEST COPY AVAILABLE 



